Entropy Regularized LPBoost
نویسندگان
چکیده
In this paper we discuss boosting algorithms that maximize the soft margin of the produced linear combination of base hypotheses. LPBoost is the most straightforward boosting algorithm for doing this. It maximizes the soft margin by solving a linear programming problem. While it performs well on natural data, there are cases where the number of iterations is linear in the number of examples instead of logarithmic. By simply adding a relative entropy regularization to the linear objective of LPBoost, we arrive at the Entropy Regularized LPBoost algorithm for which we prove a logarithmic iteration bound. A previous algorithm, called SoftBoost, has the same iteration bound, but the generalization error of this algorithm often decreases slowly in early iterations. Entropy Regularized LPBoost does not suffer from this problem and has a simpler, more natural motivation.
منابع مشابه
Di erent Boosting Algorithms and Underlying Optimization Problems
Boosting is an ensemble based method which attempts to boost the accuracy of any given learning algorithm by applying it several times on slightly modi ed training data and then combining the results in a suitable manner. The boosting algorithms that we covered in class were AdaBoost, LPBoost, TotalBoost, SoftBoost, and Entropy Regularized LPBoost. The basic idea behind these boosting algorithm...
متن کاملA Duality View of Boosting Algorithms
We study boosting algorithms from a new perspective. We show that the Lagrange dual problems of AdaBoost, LogitBoost and soft-margin LPBoost with generalized hinge loss are all entropy maximization problems. By looking at the dual problems of these boosting algorithms, we show that the success of boosting algorithms can be understood in terms of maintaining a better margin distribution by maxim...
متن کاملA Column Generation Algorithm For Boosting
We examine linear program (LP) approaches to boosting and demonstrate their efficient solution using LPBoost, a column generation simplex method. We prove that minimizing the soft margin error function (equivalent to solving an LP) directly optimizes a generalization error bound. LPBoost can be used to solve any boosting LP by iteratively optimizing the dual classification costs in a restricted...
متن کاملA non-extensive maximum entropy based regularization method for bad conditioned inverse problems
A regularization method based on the non-extensive maximum entropy principle is devised. Special emphasis is given to the q = 1=2 case. We show that, when the residual principle is considered as constraint, the q = 1=2 generalized distribution of Tsallis yields a regularized solution for bad-conditioned problems. The so devised regularized distribution is endowed with a component which correspo...
متن کاملEfficient Multiclass Implementations of L1-Regularized Maximum Entropy
This paper discusses the application of L1-regularized maximum entropy modeling or SL1-Max [9] to multiclass categorization problems. A new modification to the SL1-Max fast sequential learning algorithm is proposed to handle conditional distributions. Furthermore, unlike most previous studies, the present research goes beyond a single type of conditional distribution. It describes and compares ...
متن کامل